Lab 6: Text Classification with LSTM

This lab corresponds to Module 6 of the "Deep Learning Explained" course.

This lab shows how to implement a recurrent network to process text, for the Air Travel Information Services (ATIS) task of slot tagging (tag individual words to their respective classes, where the classes are provided as labels in the training data set).

Our model will start with a straight-forward (linear) embedding of the words followed by a recurrent LSTM. This will then be extended to include neighboring words and run bidirectionally. Lastly, we will turn this system into an intent classifier.

The techniques you will practice are:

• model building by composing layer blocks, a convenient way to compose networks/models without requiring the need to write formulas,
• creating your own layer block
• variables with different sequence lengths in the same network
• training the network

We assume that you are familiar with basics of deep learning, and these specific concepts:

• recurrent networks (Wikipedia page)
• text embedding (Wikipedia page)

Prerequisites

We assume that you have already installed CNTK. This tutorial requires CNTK V2. We strongly recommend to run this tutorial on a machine with a capable CUDA-compatible GPU. Deep learning without GPUs is not fun.

Downloading the data

In this tutorial we are going to use a (lightly preprocessed) version of the ATIS dataset. You can download the data automatically by running the cells below or by executing the manual instructions.

Fallback manual instructions

Download the ATIS training and test files and put them at the same folder as this notebook. If you want to see how the model is predicting on new sentences you will also need the vocabulary files for queries and slots

Importing CNTK and other useful libraries

CNTK's Python module contains several submodules like io, learner, and layers. We also use NumPy in some cases since the results returned by CNTK work like NumPy arrays.

In the block below, we check if we are running this notebook in the CNTK internal test machines by looking for environment variables defined there. We then select the right target device (GPU vs CPU) to test this notebook. In other cases, we use CNTK's default policy to use the best available device (GPU, if available, else CPU).

Task and Model Structure

The task we want to approach in this tutorial is slot tagging. We use the ATIS corpus. ATIS contains human-computer queries from the domain of Air Travel Information Services, and our task will be to annotate (tag) each word of a query with the specific item of information (slot) it belongs to, if any.

The data in your working folder has already been converted into a CTF (CNTK Text Format) file. Let us look at an example from the test-set file atis.test.ctf:

19  |S0 178:1 |# BOS      |S1 14:1 |# flight  |S2 128:1 |# O
19  |S0 770:1 |# show                         |S2 128:1 |# O
19  |S0 429:1 |# flights                      |S2 128:1 |# O
19  |S0 444:1 |# from                         |S2 128:1 |# O
19  |S0 272:1 |# burbank                      |S2 48:1  |# B-fromloc.city_name
19  |S0 851:1 |# to                           |S2 128:1 |# O
19  |S0 789:1 |# st.                          |S2 78:1  |# B-toloc.city_name
19  |S0 564:1 |# louis                        |S2 125:1 |# I-toloc.city_name
19  |S0 654:1 |# on                           |S2 128:1 |# O
19  |S0 601:1 |# monday                       |S2 26:1  |# B-depart_date.day_name
19  |S0 179:1 |# EOS                          |S2 128:1 |# O

This file has 5-7 columns per line (each separated by the "|" character):

• a sequence id (19). There are 11 entries with this sequence id. This means that sequence 19 consists of 11 tokens;
• column S0, which contains numeric word indices; the input data is encoded in one-hot vectors. There are 943 words in the vocabulary, so each word is a 943 element vector of all 0 with a 1 at a vector index chosen to represent that word. For example the word "from" is represented with a 1 at index 444 and zero everywhere else in the vector. The word "monday" is represented with a 1 at index 601 and zero everywhere else in the vector.
• a comment column denoted by #, to allow a human reader to know what the numeric word index stands for; Comment columns are ignored by the system. BOS and EOS are special words to denote beginning and end of sentence, respectively;
• column S1 is an intent label, which we will only use in the last part of the tutorial;
• another comment column that shows the human-readable label of the numeric intent index;
• column S2 is the slot label, represented as a numeric index; and
• another comment column that shows the human-readable label of the numeric label index.

The task of the neural network is to look at the query (column S0) and predict the slot label (column S2). As you can see, each word in the input gets assigned either an empty label O or a slot label that begins with B- for the first word, and with I- for any additional consecutive word that belongs to the same slot.

The model we will use is a recurrent model consisting of an embedding layer, a recurrent LSTM cell, and a dense layer to compute the posterior probabilities:

slot label   "O"        "O"        "O"        "O"  "B-fromloc.city_name"
              ^          ^          ^          ^          ^
              |          |          |          |          |
          +-------+  +-------+  +-------+  +-------+  +-------+
          | Dense |  | Dense |  | Dense |  | Dense |  | Dense |  ...
          +-------+  +-------+  +-------+  +-------+  +-------+
              ^          ^          ^          ^          ^
              |          |          |          |          |
          +------+   +------+   +------+   +------+   +------+   
     0 -->| LSTM |-->| LSTM |-->| LSTM |-->| LSTM |-->| LSTM |-->...
          +------+   +------+   +------+   +------+   +------+   
              ^          ^          ^          ^          ^
              |          |          |          |          |
          +-------+  +-------+  +-------+  +-------+  +-------+
          | Embed |  | Embed |  | Embed |  | Embed |  | Embed |  ...
          +-------+  +-------+  +-------+  +-------+  +-------+
              ^          ^          ^          ^          ^
              |          |          |          |          |
w      ------>+--------->+--------->+--------->+--------->+------... 
             BOS      "show"    "flights"    "from"   "burbank"

Descriptions of the above Layer functions can be found at: the CNTK Layers Reference Documentation.

Below, we build the CNTK model for this network. Please have a quick look and match it with the description above.

Now we are ready to create a model and inspect it.

Once a model is constructed, its attributes are fully accessible from Python. The first layer named embed is an Embedding layer. Here we use the CNTK default, which is linear embedding. It is a simple matrix with dimension (input word encoding x output projected dimension). You can access its parameter E (where the embeddings are stored) like any other attribute of a Python object. Its shape contains a -1 which indicates that this parameter (with input dimension) is not fully specified yet, while the output dimension is set to emb_dim ( = 150 in this tutorial).

Additionally we also inspect the value of the bias vector in the Dense layer named classify. The Dense layer is a fundamental compositional unit of a Multi-Layer Perceptron (as introduced in Lab 3). Each Dense layer has both weight and bias parameters. Bias terms are by default initialized to 0 (but there is a way to change that if you need). As you create the model, one can name the layer component and then access the parameters as shown here.

Suggested Exploration: What should be the expected dimension of the weight matrix from the layer named classify. Try printing the weight matrix of the classify layer. Does it match with your expected size?

Our input text words will be encoded as one-hot vectors of length 943 and the output dimension of our model emb_dim is set to 150. In the code below we pass the input variable x to our model z. This binds the model with input data of known shape. In this case, the input shape will be the size of the input vocabulary. With this modification, the parameter returned by the embed layer is completely specified (943, 150).

Note: As an alternative to our approach here, you can initialize the Embedding matrix with pre-computed vectors using Word2Vec or GloVe).

A Brief Look at Data and Data Reading

For reading text, this tutorial uses the CNTKTextFormatReader. It expects the input data to be in the CTF format, as described here.

But how do you generate this format? For this tutorial, we created the CTF file for you, but it might be helpful to explain how this was accomplished. The data was created in two steps:

• convert the raw data into a plain text file that contains of TAB-separated columns of space-separated text. For example:

  BOS show flights from burbank to st. louis on monday EOS (TAB) flight (TAB) O O O O B-fromloc.city_name O B-toloc.city_name I-toloc.city_name O B-depart_date.day_name O

  This is meant to be compatible with the output of the paste command.

• convert it to CNTK Text Format (CTF) with the following command:

  python [CNTK root]/Scripts/txt2ctf.py --map query.wl intent.wl slots.wl --annotated True --input atis.test.txt --output atis.test.ctf

  where the three .wl files give the vocabulary as plain text files, one word per line.

In these CTF files, our columns are labeled S0, S1, and S2. These are connected to the actual network inputs by the corresponding lines in the reader definition:

Trainer

We also must define the training criterion (loss function), and also an error metric to track the progress of our model's performance. In most tutorials, we know the input dimensions and the corresponding labels. We directly create the loss and the error functions. In this tutorial we will do the same. However, we take a brief detour and learn about placeholders. This concept would be useful for Task 3.

Learning note: Introduction to placeholder: Remember that the code we have been writing is not actually executing any heavy computation it is just specifying the function we want to compute on data during training/testing. And in the same way that it is convenient to have names for arguments when you write a regular function in a programming language, it is convenient to have placeholders that refer to arguments (or local computations that need to be reused). Eventually, some other code will replace these placeholders with other known quantities in the same way that in a programming language the function will be called with concrete values bound to its arguments.

Specifically, the input variables you have created above x = C.sequence.input_variable(vocab_size) holds data pre-defined by vocab_size. In the case where such instantiations are challenging or not possible, using placeholder is a logical choice. Having the placeholder only allows you to defer the specification of the argument at a later time when you may have the data.

Here is an example below that illustrates the use of placeholder.

While the cell above works well when one has input parameters defined at network creation, it compromises readability. Hence we prefer creating functions as shown below

Training the model

We are using the Progress Printer to display the training loss and classification error throughout training epochs.

The training should take less than 2 minutes on a Titan-X or a Surface Book. Once the training completed, you will see an output like this

Finished Epoch [10]: [Training] loss = 0.033263 * 18039, metric = 0.9% * 18039

which is the loss (cross entropy) and the metric (classification error) averaged over the final epoch.

On a CPU-only machine, it can be 4 or more times slower. You can try setting

emb_dim    = 50 
hidden_dim = 100

to reduce the time it takes to run on a CPU, but the model will not fit as well as when the hidden and embedding dimension are larger.

Testing the model

We also use the Progress Printer to display the accuracy on a test set by computing the error over multiple minibatches of test data. For evaluating on a small sample read from a file, you can set a minibatch size reflecting the sample size and run the test_minibatch on that instance of data. To see how to evaluate a single sequence, we provide an instance later in the tutorial.

This shows how learning proceeds over epochs (passes through the data). For example, after four epochs, the loss, which is the cross-entropy criterion, has reached 0.11 as measured on the ~18000 samples of this epoch, and that the error rate is 2.6% on those same 18000 training samples.

The epoch size is the number of samples--counted as word tokens, not sentences--to process between model checkpoints.

Once the training has completed (a little less than 2 minutes on a Titan-X or a Surface Book), you will see an output like this

Finished Epoch [10]: [Training] loss = 0.033263 * 18039, metric = 0.9% * 18039

which is the loss (cross entropy) and the metric (classification error) averaged over the final epoch.

On a CPU-only machine, it can be 4 or more times slower. You can try setting

emb_dim    = 50 
hidden_dim = 100

to reduce the time it takes to run on a CPU, but the model will not fit as well as when the hidden and embedding dimension are larger.

The following block of code illustrates how to evaluate a single sequence. Additionally we show how one can pass in the information using NumPy arrays.

A Word About Sequential()

Before jumping to the tasks, let's have a look again at the model we just ran. The model is described in what we call function-composition style.

Sequential([
            Embedding(emb_dim),
            Recurrence(LSTM(hidden_dim), go_backwards=False),
            Dense(num_labels)
        ])

You may be familiar with the "sequential" notation from other neural-network toolkits. If not, Sequential() is a powerful operation that, in a nutshell, allows to compactly express a very common situation in neural networks where an input is processed by propagating it through a progression of layers. Sequential() takes an list of functions as its argument, and returns a new function that invokes these functions in order, each time passing the output of one to the next. For example,

FGH = Sequential ([F,G,H])
    y = FGH (x)

means the same as

    y = H(G(F(x)))

This is known as "function composition", and is especially convenient for expressing neural networks, which often have this form:

     +-------+   +-------+   +-------+
x -->|   F   |-->|   G   |-->|   H   |--> y
     +-------+   +-------+   +-------+

Coming back to our model at hand, the Sequential expression simply says that our model has this form:

     +-----------+   +----------------+   +------------+
x -->| Embedding |-->| Recurrent LSTM |-->| DenseLayer |--> y
     +-----------+   +----------------+   +------------+

Task 1: Add a Lookahead

Our recurrent model suffers from a structural deficit: Since the recurrence runs from left to right, the decision for a slot label has no information about upcoming words. The model is a bit lopsided. Your task will be to modify the model such that the input to the recurrence consists not only of the current word, but also of the next one (lookahead).

Your solution should be in function-composition style. Hence, you will need to write a Python function that does the following:

• takes no input arguments
• creates a placeholder (sequence) variable
• computes the "next value" in this sequence using the sequence.future_value() operation and
• concatenates the current and the next value into a vector of twice the embedding dimension using splice()

and then insert this function into Sequential()'s list right after the embedding layer.

Task 2: Bidirectional Recurrent Model

Aha, knowledge of future words help. So instead of a one-word lookahead, why not look ahead until all the way to the end of the sentence, through a backward recurrence? Let us create a bidirectional model!

Your task is to implement a new layer that performs both a forward and a backward recursion over the data, and concatenates the output vectors.

Note, however, that this differs from the previous task in that the bidirectional layer contains learnable model parameters. In function-composition style, the pattern to implement a layer with model parameters is to write a factory function that creates a function object.

A function object, also known as functor, is an object that is both a function and an object. Which means nothing else that it contains data yet still can be invoked as if it was a function.

For example, Dense(outDim) is a factory function that returns a function object that contains a weight matrix W, a bias b, and another function to compute input @ W + b. (This is using Python 3.5 notation for matrix multiplication. In Numpy syntax it is input.dot(W) + b). E.g. saying Dense(1024) will create this function object, which can then be used like any other function, also immediately: Dense(1024)(x).

Let's look at an example for further clarity: Let us implement a new layer that combines a linear layer with a subsequent batch normalization. To allow function composition, the layer needs to be realized as a factory function, which could look like this:

def DenseLayerWithBN(dim):
    F = Dense(dim)
    G = BatchNormalization()
    x = placeholder()
    apply_x = G(F(x))
    return apply_x

Invoking this factory function will create F, G, x, and apply_x. In this example, F and G are function objects themselves, and apply_x is the function to be applied to the data. Thus, e.g. calling DenseLayerWithBN(1024) will create an object containing a linear-layer function object called F, a batch-normalization function object G, and apply_x which is the function that implements the actual operation of this layer using F and G. It will then return apply_x. To the outside, apply_x looks and behaves like a function. Under the hood, however, apply_x retains access to its specific instances of F and G.

Now back to our task at hand. You will now need to create a factory function, very much like the example above. You shall create a factory function that creates two recurrent layer instances (one forward, one backward), and then defines an apply_x function which applies both layer instances to the same x and concatenate the two results.

Alright, give it a try! To know how to realize a backward recursion in CNTK, please take a hint from how the forward recursion is done. Please also do the following:

• remove the one-word lookahead you added in the previous task, which we aim to replace; and
• make sure each LSTM is using hidden_dim//2 outputs to keep the total number of model parameters limited.

Works like a charm! This model achieves 0.32%, better than the lookahead model above. The bidirectional model has 40% less parameters than the lookahead one. However, if you go back and look closely you may find that the lookahead one trained about 30% faster. This is because the lookahead model has both less horizontal dependencies (one instead of two recurrences) and larger matrix products, and can thus achieve higher parallelism.